Minimizing Functionals on Normed - linear Spaces

This paper extends results of [1], [2], of Goldstein, and [3] of Vainberg concerning steepest descent and related topics. An example Is given taken from a simple rendezvous problem in control theory. The problem is one of minimizing a norm on an affine subspace. The problem here is solved in the primal. A solution in the dual is given by Neustadt [4]. I. GENERATION OF MINIMIZING SEQUENCES Let E be a normed linear space (n. 1. space), x an arbitrary point of E and f a functional defined on E. Let S denote the level set {x e E : f(x) <^ (0)} defined at an arbitrary fixed x0 e E. We denote by f'(x) the Frechet or F-derlvative of f at x. We call ^ f uniformly F-differentiable on S If f Is F-dlfferentlable 4 on S and If 5(e) In the definition of the F-derlvatlve Is constant on S. The F-derlvatlve of f at x will be denoted by f'OO. If g e E* the value of g at x will be denoted by (g,x), and If h e E** the value of h at g & E*. by Ih.g]. Recall that If E and F are n. 1. spaces and A Is bounded linear operator from E to F, In short A e B(E,F), and If A Is onto then A~ exists and belongs to B(F,E) If and only If for some m > 0 and all x e E, ||Ax| | >.m||x|| ; and that m| |xj | 0, ||g|| 1 and g(f) 0 for all f In M. Take i In E so that [g.f] [f,g] for all f In E*. Then 0 • [g,Ax] [Ax.g] for all x In E. Thus [Ag.gl 0 while ||g|| ||g|| 1. 0. E. D. Let 4» denote a bounded map from S to E satisfying the two conditions [f' (x),(|i(x)l >_ 0, and given e > 0 there exists 6 > 0 such that [f(x),^(x)] < 6 implies IJf'Cx)!! < e. Some examples of such mappings are the following: (1) Let A e B(E*,E) such that ty.Ay] io|lyl| for all 2 y e E* and some o > 0. Let ^(x) ■ Af'Cx) and choose 6 ■ e a. Then Hf'U)! | < e. As a possible candidate for the operator A, suppose f is twice F-differentiable on E. Assume that for some u > 0, and some x in S the operator f"(x) in E>(E,E*) is onto and is "oounded below", that is, the bilinear functional satisfies [f"(x)zfz] ^p||z|| for all z in E. Then | |f"(x)z| | ^ u| |z | | showing that f"(x) has an inverse [f"(x)l "Ac B(E*,E). Since A has a bounded inverse, there exists a number o > 0 such that | |Ay| | ^o||y|| for all y e E*. 2 2 Set z ■ Ay. Then lf"(x)z,z] ■ [y.Ay] ^ uo ||y|| showing the candidacy of A. (2) Suppose E is a reflexive Banach space. By the weak compactness of the unit sphere in E it follows that for some z , !|z0l| ■ It [f(x),z0] ||f'(x)||. Set ♦(x) z0||f(x)||. Because 2 [f(x),4)(x)l ■ IJf'Cx)!! , ^(x) is the analogue of the gradient in Hilbert space. When E is an L space the point z is obtained by considerations of equality in Holder's inequality. (3) Since Hf'Cx)!! sup{ [f (x) ,z]: | |z | | 1}, If 0 < a < 1 a point zexists such [f'M.z ] >^ a| | f' (x) | |. If for fixed a and all x e S we can find such z , we may take i>(x) « z ||f*(x)| |. In what follows let A(x,p) ■ f(x) f(x p4)(x)) and g(x,p) ■ A(x,p)/plf' (x) ,<|)(x)]. Assume E is a normed-linear space and S is the level set of f at x. in E. In what follows, assume 0 < o < ^. Theorem. Assume that on S f is uniformly F-differentiable or that the F-derivative f exists and is uniformly continuous. Set x. .. ■ x. , when [f' (x. ),(j)(x, )] ■ 0; otherwise choose* p. so that o < g(x.,p.) <^ 1 o when g(xk,l) < o or Pk " ! w hen g(xk,l) ^ o, and set xk+1 " xk pk^(xk). (a) If S is bounded or f is bounded below then {f'(x.)} converges to 0 while {f(x.)} converges downward to a limit, L. If S is compact, then every cluster point of {x, } is a zero of f*. In addition, if 4»(x. ) ■* 0 and f has finitely many zeros, {x. } converges. (b) If S is convex and bounded and f is convex, L « inf{f(x):x e S} » 6. If, in addition, E is a reflexive Banach space, then every weak cluster point of {x. } minimizes f on E. If E ■ L [0,1], k p then {x. ^ converges to a unique minimizer of f. (c) Assume that the Gateaux derivative f" exists on S and satisfies i2 . „„. x_ . . ..,,,,2 u\ |z| i" <^ (f(x)z,zj 1 M| |z| I for all x e S, z e E and some u > 0. Assume S is convex and E is complete. Then (x, } converges to a unique minimizer of f on E. *If the Gateaux differential f" satisfies f(x,h,h) <. ||h||/p for all h in E, x in S and some p. > 0 we can choose p. to satisfy 6 ^ p, ^ 2p 6 with 0 < <5 ^ P0. The method of steepest descent could also be employed. See [9]. The proof of (a) is given in [1J. The proof there is stated for E, a Hubert space, but the same proof works when E is taken to be a n. 1. space. Two comments might be made, however. S bounded and f uniformly continuous on S implies that f is bounded on S. (See e.g. [5], p. 19.) It follows by employing the mean value theorem that f is bounded below on S. The statements f uniformly F-differentiable and the F-derivative f is uniformly continuous are equivalent. (See [3], p. 45.) (b) Given e > 0 choose z' e E such that f(z) ■ 6 + r/2. Because f exists at x. and f is convex, f^*) ^ *(*].) "♦" lf(l-)» ~ iJ • Since {ffa. )} ■* 0 and S is bounded, for all k sufficiently large f(x. ) <_ fU') + e/2 ■ 6 + e, showing that L ■ 0. If E is reflexive and S is convex, closed, and bounded, then S is weakly compact. Since f is convex, the sets (x e E:f(x) <^ k} are closed, convex, and weakly closed, for all k. Thus f is weakly lower semi-continuous. If z is a weak cluster point of {x. } then for an appropriate subsequence, lim inf f(x.) ■ L > f(z). Assume E » L [0,1] k. p and f is the norm on E. By [6], p. 78, if {x. } converges weakly to z and f(k) "* z then {x. } converges strongl) to z. It follows that every weak cluster point of (x. } is a strong cluster point of (x. }. Since f* vanishes at every weak cluster point of (x.} and f vanishes only once by the strict convexity of f, every subsequence of (x. } has the same weak cluster point z, showing that {x. } converges to z. (c) The hypothesis of (c) Imply that f Is Lipschitz continuous and that the set S is bounded. Otherwise S would contain an unbounded sequence, say {z. By Taylor's theorem if u e S. f(2k) >_ f(u) + | |zk u| |[(| |zk u| |y/2 -Hf'Cu)! |j, showing that f(k) 1 f(x0), for large k, whence S must be bounded. We now show that the sequence {x. } is Cauchy. Again by Taylor's theorem if s > k, f(xg) f(xk) >. [f'(xk). xs xkJ +u||xs xk|| /2. Since S is bounded, | |x x. || £ D where D is the diameter of S. Thus ll Xull l"^f(x ) f(x.) + D||f'(x.)||} which shows that {x} is a Cauchy sequence. By the completeness of E (x } has a limit, say z, in E, and f (z) ■ 0. If z is not unique, t>3n f'Cz.) f iz-) • 0, z. # x. Thus f(.z.) fCzj) >, ■oil*! «oil 1 f ^2^ " ^i)« ■ contradict ion. Hence z is unique and is a minimizer of f. Remarks; Useful remarks may be found in [1], [3] and [9]< II. NEWTONIAN STEPS AND ACCELERATION Suppose that at the given point x0, the function f satisfies the conditions of the first example, namely 4>(x) ■ f"(x0)f'(x), where f"(xn) If'Cx-)]" . The corresponding iteration is x ^ x p f"(xjf'(x ). _i u u n+i n n u n This algorithm, when p ; 1 is known as the "modified" Newton's method (see n [3], p. 239, or [7], p. 696). In a similar manner if f"(x) exists and is uniformly bounded below on S, ve may define $(x ) ■ f'Cx )f'(x ). We n -l n n shall do this below. It is clear from what has already been said that ^ satisfies hypotheses of the above theorem. Our object now is to formulate an algorithm using f'Cx )f(x ) . m| |z | | 2 for some m>0 and all z in E. Set x. . " x. p.f'^x^f'Cx, ), where p, is chosen so that for e < 1/2, 0 < 6 1 g(x. .p.) <_ 1 9 with Pk " 1 if possible. Then; (a) There exists a number N such that if k > N then p. ■ 1. (b) There is a unique minimizer of f and the sequence (x. } converges to it faster than any geometric progression. Proof, We have for all x in S that M| |z | | 2 >_ [f (x)x,x] >_ m| |z | | 2 and m^Hyll > [y.f'(x)y] im>r||y||. Thus if <Hx) r(x)f(x), -2 2 then (f(x),^(x)] ^ mM llf'Cx)!! , showing that $ satisfies the conditions of the above theorem. Since f" is bounded on S, f is Lipschitz continuous, by the mean value theorem. By (c) above (x. } converges to a unique minimizer of f. Expand A(x,p) to two terms in the Taylor series with remainder [f( )h,h), where h = p^(x)f(x). Set f"(0 f"(x) + f'CO f'U). Ther g(x,p)-l -r/2p[(f"(C) f^x) )f[(x)f ' (x) ,f|(x)f' (x) ]/2[f' (x) ^'(x) f ' (x) ] _ 1 ,72 pi|f"(0 E"(x)| lM/2m. Thus |g(x,p) 1 + P/2| 1 p||f(0 f"(x)|lM /2m. Since f(p, ) lies between x. and x. ., x/,,^,x. , r ,. .. k k k+1 0 0 1 is a Cauchy sequence; and It, together with its limit z, is a compactum. Consequently, on this compactum f" is uniformly continuous, so that {||f"(C(p,) f(x)||} converges to 0, showing that the choice p « 1 is eventually feasible. To prove (b) we write x. . z ■ x, z p.f "(x. )f' (x. ) = xk z pkf , i (xk)f"(xk)(xk z) + pkf'i'(xk)[f"(xk)(xk z) f'(xk)]. Thus ||xk+1-z|| |]xk-z-pk(xk-z)|| +Pk||f' (xk)||||f (xk)(xk-2)-f(xk)I|. Since f is F-differentiable at xk, \\{'(z) *'(\) ' f"(xk)(z xk) | | < e | |z xk||. Thus ||xk+1 z|| » (1 pk)| |xk z|| + Pkm e||z xk|| Q.E.D. Remarks; (1) Both sides of the inverse of f"(x) are used in the proof. (2) The analogue of the modified Newton process, namely choosing <|i (x) f"(xJf'Cx) or f"(x.)f(x) with k fixed also will under the hypothesis —1 u _^ k of the above theorem generate a sequence converging to a unique mlnimizer of f. Since l|xk+1-z|| | Ix^z-P^x^r (z) (x^z) | | + ^1 I (£'^(XQ) ] I I ^ ||x.-z||m when | |x -z | | < 6, the rate of convergence is eventually geometric provided ||l p f"(x )f"(z) | | < 1. Since ||i pkf^ (x0)f '(Z)|| < i pk + pkMr(x0)|| ||(f"(x0) f"(z)||, if llf'Cx ) f"(z)|| is sufficiently small, p = 1 will generate a sequence converging to z at the rate of geometric progression. A sufficient condition for the global geometric convergence would be (M/m) < 1/2, since -1 (3) Pertinent remarks may be found in [3). ||f(x)|| IM and ||r(x)|| im" .